Hierarchical Sparse Coding

A number of researchers have theorized that the brain may be employing some form of hierarchical model of features in visual processing. Nodes at the bottom of the hierarchy would represent local, spacially-oriented, specific features, while levels further up the hierarchy would detect increasingly complex, spatially-diffuse, and invariant features, with nodes in the uppermost layers corresponding to invariant representations of objects and concepts. For example, Mumford and Lee have outlined such a system employing hierarchical Bayesian inference to combine sensory input at the lowest levels with feedback from priors higher up [7]. Models have been developed based on the idea of sparse coding that seem to mimic many of the observed features of area V1 in the visual cortex—the lowest layer of the hierarchy. Specifically, we assume that natural images can be represented as a sparse linear combination of over-complete basis functions. Using unsupervised learning techniques and optimizing for sparseness, Olshausen and Field succeeded in generating such a set of bases that resemble the localized, oriented lines detected by simple cells in V1 [8]. Bell and Sejnowski used independent component analysis (ICA) and the infomax principle—maximizing the information preserved by the decomposition—to produce bases with similar characteristics [1]. These models are good as far as they go, but they cannot be readily extended to generate higher layers. In particular, we have assumed that the data is a linear combination of independent components, which limits the complexity of the structure that can be captured. Simply generating a new sparse code for the output of the first layer yields no new information.